Add Anthropic provider (apache-airflow-providers-anthropic)#69003
Conversation
kaxil
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl and
potiuk
as code owners
boring-cyborg
Bot
added
area:dev-tools
area:providers
backport-to-v3-3-test
kaxil
force-pushed
the
add-anthropic-provider
branch
2 times, most recently
from
3b8adcc to
b64bf04
Compare
amoghrajesh
left a comment
amoghrajesh
left a comment
There was a problem hiding this comment.
From a static code review, I am taking it for a spin now to see how it goes.
| ENDED = "ended" | ||
|
|
||
| @classmethod | ||
| def is_in_progress(cls, status: str) -> bool: |
There was a problem hiding this comment.
This makes sense, but I think it's a bit confusing.
in_process is in_process, which is expected, but canceling also is in_process. It's not incorrect, though it does make the meaning of in_process less clear.
There was a problem hiding this comment.
Fair point. canceling is non-terminal (we poll until ended), so it returns True too, which reads oddly next to the in_progress enum value. The method really means "not yet terminal"; I kept the name to mirror the SDK's processing_status vocabulary. Happy to rename it to is_not_terminal (or add a docstring note) if you'd prefer, let me know.
There was a problem hiding this comment.
Consistency sounds like a better way to go. But let's extend the docstring/comment here so we can avoid confusion as much as possible. Thanks
kaxil
force-pushed
the
add-anthropic-provider
branch
from
b64bf04 to
260a157
Compare
Add a dedicated provider for the Anthropic Claude API so Dag authors can use Claude through the official Anthropic Python SDK instead of hand-writing SDK calls in tasks. It offers first-class connection management (first-party API, Amazon Bedrock, Google Vertex AI, Claude Platform on AWS, Microsoft Foundry, and keyless Workload Identity Federation) with labelled connection form fields, plus deferrable Message Batches and Managed Agents session operators, a batch sensor, and matching triggers. Targets Airflow 3+ and ships as not-ready / incubation while the Managed Agents beta surface stabilises.
kaxil
force-pushed
the
add-anthropic-provider
branch
from
260a157 to
6cbb964
Compare
kaxil
requested review from
choo121600,
ephraimbuddy,
jedcunningham and
vatsrahul1001
as code owners
3 participants
New provider
apache-airflow-providers-anthropicfor the Anthropic Claude API, so Dag authors call Claude through the official Anthropic Python SDK instead of embedding SDK calls in task code.It ships:
AnthropicHook— builds the right client for the configured platform: first-party API, Amazon Bedrock, Google Vertex AI, Claude Platform on AWS, and Microsoft Foundry. Supports keyless Workload Identity Federation (configured on the connection or resolved from the environment).AnthropicBatchOperator(deferrable) +AnthropicBatchSensor+AnthropicBatchTrigger— the Message Batches API (submit → defer → collect).AnthropicAgentSessionOperator(deferrable) +AnthropicAgentSessionTrigger— Managed Agents sessions (message and outcome runs).conn-fields, not raw Extra JSON.Design rationale
common.ai.common.aiis a provider-agnostic abstraction; this provider exposes Anthropic-specific surfaces (Message Batches, Managed Agents, token counting, the platform clients) that do not fit a neutral interface. Users who want a provider-agnostic layer keep usingcommon.ai; users who want direct Claude/SDK features use this one.deferrable=Falsemirrors the same cancel-on-timeout (batch) / archive-on-timeout (session) teardown so neither mode leaks a server-side resource.modelon the connection. The default model is read fromextra['model']so it changes without editing DAGs; it falls back toclaude-opus-4-8.Notes / gotchas
platformin {anthropic, aws}; calling them on Bedrock/Vertex/Foundry raises a clear error rather than a raw 404.state: not-ready/lifecycle: incubationwhile the Managed Agents beta surface stabilises.anthropic>=0.101.0(introducedAnthropicAWS); verified against 0.109.x.Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.